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Abstract 

The use of online learning is prevalent in higher education, in particular the use 
of Blackboard and other similar virtual learning environments (VLEs). Students' 
participation in online discussion forums is a common activity and the online 
threaded discussion tool provides opportunities for students to: 

• form collaborative groups for problem-based learning 

• discuss learning experiences 

• ponder over assessment issues 

• debate topics of interest. 

There is no shortage of research on the perceived effectiveness of online discussion forums; 
however, this paper evaluates the potential for assessing the content of online discussions as 
a means of contributing to summative assessment. 


Introduction 

The use of online learning is prevalent in higher education, in particular within virtual learning environments 
(VLEs), such as Blackboard; participation in online discussion forums is a common activity. Considerable 
research exists on the potential of using the discussion board for socialisation (Fauske and Wade, 2003), 
discussing learning experiences (Hammond, 2000), assessment issues, and debating topics of interest 
(Laurillard, 2003). Within my institution, discussion boards have been used as a collaborative tool across 
a range of information communications technology (ICT) modules to promote discussion and reflection. 

The nature of the subject often requires a disproportionate amount of time developing skills in using 
software tools, to the detriment of discussion and reading time. 

Opportunities for wider reading and discussion are crucial to the learning and they afford opportunities 
for students to discuss and question key research in the field, by observing the contributions of others and 
offering alternative viewpoints. As student commitment and co-operation is crucial to this process, it was 
hoped that by making discussion board participation an assessed element, student engagement would 
increase in both quantity and quality. This feasibility study therefore looks at the issues that arise in applying 
a content analysis tool (the Newman model) to assess students' contributions to online discussion forums. 

Numerous publications exist on the benefits of eLearning, online learning and use of computer mediated 
discussion (Salmon, 2000), but information focusing on analysing and assessing discussion threads is not so 
well researched (Hora and Kling, 1999). Existing research in health-related journals focuses on discussion board 
use as a tool for initiating discussion and engaging students in problem-based learning scenarios (Braidman et 
al., 2008), where the participation in online discussion is qualificatory (some contribution is required to obtain 
a pass), and not contributory (counts towards the final grade awarded). Analysing discussion postings for 
purposes of assessment that would derive quantitative results requires a specialised tool, some of which are 
discussed here. 
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Within the subject of ICT education, in my institution, many assessments comprise coursework portfolios 
and/or the development and creation of a physical resource with accompanying narrative. Historically, student 
teachers have often focused on skills development of learning new software for the portfolio resource, rather 
than engaging with the wider reading about how the created resource contributes to pupil learning. Despite 
weekly discussion-board topics being set up, students often fail to participate in this element of the course, 
and thus arrive at sessions ill-prepared to contribute to the ongoing discussions. The revalidation of the ICT 
education degree courses provided an opportunity to take a fresh look at assessment issues, and the idea of 
piloting students' postings to the discussion board as a compulsory assessed element, was considered. 

The research and review of literature 

Considerable research has been undertaken on how the use of a discussion board can be used more 
productively to contribute to learning and teaching (Benigo and Trentin, 2000; Meyer, 2003), but little has 
been written on how it can inform course assessment. Encouraging students to participate in online discussion 
is problematic, but the use of effective questioning techniques (scaffolding the discussion) has been shown 
to elicit higher-quality responses and fewer off-task postings (Cho and Jonassen, 2002; Wang, 2005). Others 
perceive this scaffolding as enforcing a constrained forum and directing students down one particular path 
(Rush, 2004). Wang's study (2005) demonstrated that the use of effective questioning techniques elicited 
better-quality responses from students and also encouraged higher-order thinking. 

Similarly Im and Lee's (2004) topic-related discussion resulted in substantial learning because students had 
time to reflect before participating in online discussion forums. Garrison, Anderson and Archer (2001) also 
supported the use of online discussion as being significant in developing critical thinking skills, resulting in a 
more structured forum than face-to-face communications. There is no doubt that as a tool for socialisation, 
the use of online discussion boards allows students extended opportunities to contribute to discussion 
ideas (Salmon, 2000; Garrison et al., 2001). The concern, however, is that these 'results' may be viewed as 
subjective, unless the data can be analysed with a tool that would make the system transparent. 

Early work in the field (Henri, 1992) analysed the content of online postings by measuring four factors: 

• social 

• interactive 

• metacognitive 

• cognitive. 

with the cognitive factor being further broken down into five types of reasoning skills. Fisher (1997) identified 
types of discussion as: 

disputational, cumulative and exploratory 

resulting in the 'dialogical' framework as described by Wegerif and Mercer (1997). More recently, Riley (2006) 
argued that his: 

. ..content analysis of the online discussion through a dialogical framework for social modes of thinking to 
measure social reasoning... 


and 

...keyword identification and concept mapping connectivity to measure conceptual transfer from the 
online discussion to individual students and higher order thinking... 

(numerical frequencies of participation, talk type and triangulation of data using concept mapping) could 
be applied as an evaluative tool to assess the quality of the discussion exchange, by assigning a 'talk type' 
from the dialogical framework. However, as with many other methods, the qualitative measure cannot easily 
produce quantitative results for the purpose of summative assessment. The amount of subjectivity involved 
also has to be taken into account. Riley's study commented upon: 

. ..determining the progression of thinking and social interactions taking place online... 

but a flaw in this study was the mechanism by which Riley extracted his quantitative data. Using the 
AutoSummarise tool in Microsoft Word he identified and determined key points and key words by analysing 
documents and assigning a score to each sentence. If the 'key words' were used often, then the score 
was increased. 
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Research by Ahern, Peck and Laycock (1992) studied the whole message, assessing the interaction levels and 
complexity of response. Zhu (1997) looked at the whole theme of the message, with the elements of: 

• participation 

• social 

• cognitive 

• metacognitive 

being the variables investigated, but the reliability of those studies is not reported. Mason (1992) in her study, 
reported on the data derived from the VLE-generated statistics, outlining the number of postings per student 
and the number of times each was read in the overall discussion forum. Ascertaining the quality of the 
postings, however, would be difficult to achieve using any of these methods. 

Methodology 

Preliminary discussions were conducted with 20 ICT subject specialist student teachers across a range of 
programmes: 

• undergraduate primary (pupils 4-11 years) 

• undergraduate secondary (pupils 11-18 years) 

• postgraduate ICT courses 

about the feasibility of assessing their contributions to the discussion board as part of the summative 
assessment process. To ascertain 'what if?' discussion-board content contributed to 15% of the final 
assessment mark, discussion on the feasibility of assessing online discussion content ran concurrently with 
other discussion topics on Blackboard. The first forum posted to all module discussion boards consisted of the 
following statement: 

Online discussion is considered to be a suitable medium to discuss key readings prior to face-to-face 
sessions. As this is part of the learning process, should your contributions be assessed or not? 

The second forum was a structured question relating to the pre-reading for the module. Students were given a 
research paper (Ogborn, 1998) and the question posed was: 

Is computer-based modelling more viable as an individual activity or as a group activity? Argue the case 
for group activity. 

Students used questioning techniques to facilitate initial discussion and were invited to consider the 
assessment of each other's postings by applying the Newman content-analysis tool to gain an understanding 
of how a raw mark could be derived for assessment purposes. They were reassured this was primarily an 
experiment to determine how easily the content could be graded. 

From the outset, students were aware that they were part of a pilot in determining the feasibility of assessing 
the content of their online discussions using a quantitative tool. The student cohorts agreed to have their data 
analysed using a range of tools for this small-scale study. Initially, this was going to be tutor led, with the tutor 
applying the tools and deriving a 'mark' dependent on the results of the data analysed. After many attempts 
at applying various tools, the plan was amended and students were asked to peer review the data, thus 
allowing them to develop an understanding of the transparency of the process. 

For the purpose of deriving a mark for the content, it was evident that a tool that could be applied in a 
quantitative way was required. Two tools that appeared to be fit for purpose were the Interaction Analysis 
Model (1AM) (Gunawardena, Lowe and Anderson, 1997) and the Critical Thinking Model (Newman, Webb and 
Cochrane, 1997). Although Newman's original model was fairly dated it has been extended by Kirschner et al. 
(2004) who proposed four elements: 

• contribution 

• verification 

• clarification 

• elaboration. 

Unfortunately, although this version was simpler to use, it did not provide the necessary transparency to award 
a grade. With the original Newman model, a formula could be applied and a result calculated. 
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Following the debate on computer modelling as a group activity, and subsequent postings to the discussion 
board, students applied the analysis tool to the content by selecting ten categories as defined by Newman: 

1 relevance 

2 importance 

3 novelty 

4 outside knowledge 

5 ambiguities 

6 linking 

7 justification 

8 critical assessment 

9 practical utility 

10 width of understanding. 

For each statement or paragraph, students had to apply a plus (+) or minus (-) factor to specific statements. 


Figure 1 Subset of Newman et al. codes (1996) 


Category complete list 

Relevance; importance, Novelty, Outside knowledge, Ambiguities, Linking, Justification, Critical assessment, 
Practical utility, Width of understanding 

Category rating examples 

Positive indicator 

Negative indicator 

R+ Relevance 

R+ Relevance statements 

R- Irrelevance statements diversions 

1+ Importance 

1+ Important points/issues 

1- Unimportant, trivial points/issues 

N+ Novelty, new info, ideas, 
solutions 

NP+ New problem-related 
information 

NP- Repeating what has been said 


NI+ New ideas for discussion 

Nl- False or trivial leads 


NS+ New solutions to problems 

NS- Accepting first offered solution 


NQ+ Welcoming new ideas 

NP- Squashing, putting down 
new ideas 


NL+ Learner brings new things in 

NL- Dragged in by tutor 


The decision to concentrate on the Newman model afforded a calculation feature that allegedly measured the 
quality of the data and not the frequency of postings as in other methods. The ten categories also had coding 
subsets to further exemplify the data if required. By applying a code to each sentence or paragraph, the 
positive indicators and negative indicators could be calculated and results obtained. 

Results 

The results from this study illustrated that certain categories were easy to identify, namely relevance, linking and 
justification. Very few students identified critical assessment and some identified difficulty with some of the 
meanings, for example - practical utility and outside knowledge. If outside knowledge was applied, did this 
mean that it couldn't be graded for novelty1 If someone posted a novel idea, could this be incorporated in 
width of understanding! If you widen knowledge does this mean that you slip into the outside knowledge 
category? It was difficult to award a negative rating, as the tool could only be applied if the code category 
was present. It was therefore not possible to state that an element of the discussion was, for example, not 
ambiguous. A coding could be applied if something was ambiguous, but if it wasn't, it was likely to be either 
relevant, important, novel or outside knowledge. The whole process, while interesting, was also extremely 
time consuming and although this appeared the most useful tool for analysing the content, the code categories 
available did not cover all of the options that the students felt were present within specific postings. The formula 
applied to the coding comprised the mean of the number of positive and negative contributions, for example 
27 J+ codes and 11 J- codes would result in a critical thinking ration for the justification category of 0.16. 
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Conclusion 

Using an asynchronous discussion tool as a means of providing quantitative results for assessment purposes 
is fraught with difficulty, none less than the strategic stance learners might take in 'writing for the grade' 
(Elbow, 1997). Statistical data counting the frequency of postings can be easily analysed. The validity of 
this data, however, is problematic. Student success could not, and should not, be defined to the number 
of postings, but rather to the quality of the content of those postings. 

The Newman model, while the most appropriate tool to date for this task, still resulted in considerable 
ambiguity. Students analysing their own postings had a clear indication of what they meant and had no 
difficulty with their own analysis. They were however troubled with the thought that this could be used to 
derive a grade for assessment purposes, as often their interpretation of other students' postings were difficult 
to quantify and, by their own admission, were often subjective. The task of analysing the data was onerous 
and resulted in a wide variety of opinion. Peer agreement of categories was achieved in only three of the ten 
categories, but the amount of time and effort to derive this result was onerous. 

While all the tools provided different types of results, concerns remained about the lack of any mechanism 
to analyse the literacy aspect within the postings. If these discussions were to form part of the summative 
assessment, then surely attention to detail (grammar, spelling, sentence construction) would also have to 
be taken into account? Interestingly enough, part of what is encouraged within higher education (good 
academic writing; attention to detail; structure) may be largely ignored in an online environment. 

Messages (or similar) stating: 

... don't worry about spelling or grammar when using the online environment... 

often appear as introductions to online activities. Why should the discussion board environment be any less 
important if it is to contribute to the overall learning and teaching process, and to assessment in particular? 

It could be argued, however, that the mulling over grammatical detail could detract from the spontaneous 
use of the discussion board. 

This is a 'work in progress', but the indications are that it is not feasible to derive a 'mark' for such an activity. 
Taking into account the literacy issues, quality of content (where students determine the criteria factors and 
have added substantially to the Newman categories), quantity of initial posting and responses to other 
postings, there is still some way to go in developing an ideal tool fit for purpose. While 30% of the students 
would welcome credit for their contributions to online discussions, the remainder demonstrated a variety 
of concerns; namely the cumbersome workload, the level of subjectivity, and the fact that they could never 
be fairly graded. All students conceded, however, that the use of the discussion tool enhanced session 
participation and were in agreement about pursuing the research further. 

As a cognitive tool, the discussion board affords time for the learners to: 

• reflect 

• share ideas 

• learn through observing and interacting with others 

• create their own view of the topic. 

Isolating a component part of using the VLE and the discussion board tool, in particular for assessment 
purposes, appeared to detract from the main purpose of engaging with course content to foster 
deeper learning. 

If one advantage of using discussion boards is to assist in the process of learning, it raises the question 
(for future research) whether it is appropriate to assess the content of those online discussions at all. 
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